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Abstract 

In Random Matrix Theory the local correlations of the Laguerre and Jacobi Unitary 
Ensemble in the hard edge scaling limit can be described in terms of the Bessel kernel 

Ba{x,y) = ^/xy ^ > a;,y>0, q > -1. 

In particular, the so-called hard edge gap probabilities P^"\R) can be expressed as 
the Fredholm determinants of the corresponding integral operator Ba restricted to the 
finite interval [0, i?]. Using operator theoretic methods we are going to compute their 
asymptotics as i? — ?■ oo, i.e., we show that 

:= det(/ - S„)U2[o,«] ~ exp ( - + «i? - ^ logi?) 

where G stands for the Barnes G-function. In fact, this asymptotic formula will be 
proved for all complex parameters a satisfying |ReQ| < 1. 

1 Introduction 

For 1 < p < oo and a subinterval / of the real line, let Lp{I) stand for the usual Lebesgue 
spaces. We denote by the integral operator acting on L^(]R+), R_|_ = (0, oo), with the 
kernel 
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where Ja{x) are the Bessel functions with (complex) parameter a. We will always assume 
that Rea > —1. 

In this paper we will be concerned with the Fredholm determinant of Ba on the interval 
[0, i?], i.e., with the quantity 

p(")(/2) := l + y^- — r- / ••• / Bo,{xi,Xj) dxi...dxn. (2) 

^ nl Jo Jo L -li,i=i 

An equivalent definition can be given as an operator determinant [20l 135] , 

P(")(i?) := det(PR - PRB^Pn)\mo,R]- (3) 

Here Pr stands for the orthogonal projection from onto the subspace L^[0, R], 

T-. r/ \ / \ f fix) iiO<x<R 
Pn--m^9{x) = ifx>P." 

We remark that the operators PR-BaPR|L2[o,R] are trace class operators on L'^[0,R]. 

The goal of this paper is to determine the asymptotics of P^"\R) as i? — i- oo. In fact, 
we will prove the asymptotic formula 

P(") (P) ~ exp ( - ^ + aP - y log P) ^1^, ^ ^ oo, (5) 

under the assumption |Rea| < 1. Herein G stands for the Barnes G-function fl], which is 
an entire function defined by 

oo 

G{1 + z) = (27r)^/2e-(^+^)^/2-7^.V2 "Q (^(^ + z/k)'e-'+'"/^^''^^ (6) 

k=l 

with 7E being Euler's constant. 



For real a > —1, the quantity P*^")(P) appears in Random Matrix Theory [251 [371 [Ml HQ] 
as the gap probability for certain random matrix ensembles in the so-called hard edge scaling 
limit. Recall that the gap probability is the probability that no eigenvalue of the random 
matrix lies in an interval of some length. The Bessel kernel Ba{x,y) arises, for instance, as 
the correlation function in the hard edge scaling limit of the Laguerre and Jacobi Unitary 
Ensemble (LUE/JUE) [HI [261 [27] as well as of generalized LUEs and JUEs [231 [H]. All 
these ensembles consist of complex self-adjoint matrices equipped with a certain probability 
measure which is invariant under unitary transform. 
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To be more specific, recall that the LUE consists of positive self-adjoint complex N x N 
random matrices such that the joint probability density function of the eigenvalues is given 
by 

N 

Pa^^'l'^lj-'-'-^A^) = Ca,Ar]^Afce"^*= Y\ \\j - Xkf , Ai, . . . , Aat > 0. 

k=l l<j<k<N 

The JUE consists of all contractive self-adjoint complex N x N random matrices with joint 
probability density function of the eigenvalues given by 

pS,iv(Ai,...,A^) = c,,^,jvn(l-^fc)"(l + ^fc)^ n I^J-^fcl'' -1< Ai,...,A;v < 1. 

fc=l l<j<k<N 

In both cases and for finite N, the probability that no eigenvalue lies in a subinterval / of 
]R+ or [—1, 1], respectively, can be written as a Fredholm determinant 

\k 





./det 











dxi ■ ■ ■ dxk- 

l<r,s<k 



k=l 

Here stands for the orthogonal projection onto the subspace of L^(M+) or 1,1], 
resp., spanned by the first Laguerre or Jacobi functions, respectively, and KN{x,y) is the 
integral kernel of this operator. 

In the random matrix context, a and P are real parameters greater than —1. Let 
pLag,(a)^^^j and P^^'^"'^\r) denote the probabilities that no eigenvalues lie in the inter- 
val [0,R] C IR+ (Laguerre case) or [1 — R,l] C [—1,1] (Jacobi case), respectively. Notice 
that these probabilities describe also the behavior of the eigenvalue closest to the hard edges 
of the ensembles. With the appropriate scaling these probabilities converge (as N — )■ oo) to 
the quantity P(")(i?): 

P(")(i?) = lim P^ag,H/JR^^ _ ij pJ-.(-./3) J^A 

In fact, the Laguerre and Jacobi kernel converge, after the hard edge rescaling, to the Bessel 
kernel Ba{x,y). This is also true for certain modified Laguerre and Jacobi random matrix 
ensembles [23l HU . 



Usually the kernel 

Ba{x,y) = -J ^ 

2{x - y) 

is referred to as the Bessel kernel. It relates to our Bessel kernel by a simple change of 
variables. In particular, 

P(")(yp) = det(PK - P/?P.Pi?)|L2[o,^]. 
Our form of Bessel kernel is more appropriate for the computations in this paper. 
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The asymptotics ([5]) was conjectured by Tracy and Widom [39]. We conjecture that it 
is vahd for all complex a with Re a > —1, although our proof works only for |ReQ;| < 1. 
Following [39], let us explain what has supported the Tracy- Widom conjecture. If we define 

a(.) = -.^(logP(-)(v/^)), 

then it is known that a satisfies the differential equation 

{sa"f + a\o - sa')(4(T' - 1) = o?{a'f, (7) 



which is reducible to a Painleve III equation with specific parameters (see (3.13) in 
and p8]), and, in fact, also to a Painleve V equation. (The equation ([7]) is essentially the 
Jimbo-Miwa-Okamoto Painleve III' form.) Based on this differential equation, one can make 
a guess that the specific solution has an asymptotic expansion as s — )■ oo in powers of s^^^, 
i.e., (t(s) = c\s + C2S^I'^ + C3 + C4S~^/^ + . . .. A straightforward computation yields recursively 
the coefficients (up to some ambiguity in a sign), and one obtains 

2 

^(^) = \- r\s^'^ + ^ + + . . . , s ^ 00, (8) 

with either r = lorr = — 1. The actual value of r must be determined differently. Finally, 
upon integration one obtains the asymptotics (I5l) for P^°'\R), and even higher order terms, 
except for the constant term (27r)~"/^G(l + a). 

The conjecture [39] for the constant term (as well as for the correct value r = 1 in ([8])) 
relied on the special cases a = ±1/2. In these two cases, the Bessel operator equals a 
Wiener-Hopf-Hankel operator, 

sin(x - y) sin(x + y) 

B±i/2{x,y) = ^ T —f — \ — ^, 

'Kyx — y) 7r(x + |/j 

and the asymptotics, including the constant, was conjecturally identified by Dyson [14J 
based on rigorous work of Widom [12]. The rigorous identification of the constant (and in 
fact a proof of the asymptotics ([5])) in the cases a = ±1/2 was given by the author [16] by 
employing the same kind of methods that will be used in the present paper. Another proof 
for the special cases a = ±1/2 based on Painleve transcendents and using the Riemann- 
Hilbert method [13] was given recently by Baik, Buckingham, DiFranco, and Its |9J. The 
quantities P'^^^/^)(i?) also occur in connection with the gap probability in the bulk scaling 
limit of the Gaussian Orthogonal and Symplectic Ensemble [23,110] ■ 

So far the derivation of ([8]) is only heuristic, and the author is not aware of a rigorous 
proof. However, let us make the following observation. Because it is comparatively easy to 
determine the asymptotics of a{s) as s — )■ 0, the solution of the so-called connection problem 
for the Painleve equation ([71) would provide such a proof. Unfortunately, the author did not 
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find a rigorous solution of tfie connection problem for this concrete Painleve equation. Thus 
this problem still seems to be open. However, considerable progress has been made in solving 
connection problems for other classes of Painleve equations. The techniques employed there 
appear to be powerful enough to solve also the connection problem for (^^. We will not 
discuss this topic any further, but refer the interested reader to the monograph of Fokas, 
Its, Kapaev, and Novokshenov [18], as well as to the various articles of Kitaev, Andreev, 
Vartanyan, and Novokshenov referenced in this book. 

There exist similar "gap probability problems", involving different kernels and arising 
from different scalings and/or random matrix ensembles. These problems were open for 
quite some time and were solved only recently. Besides the special case a = ±1/2 already 
mentioned above, the perhaps most important one was the problem of asymptotics of the 
Fredholm determinant of the Wiener- Hopf sine kernel, sin(a; — y)/{7i{x — y)). This quantity 
describes the gap probablity for the Gaussian Unitary ensemble in the bulk scaling limit. 
The asymptotics was rigorously determined by Krasovsky [23] using the Riemann Hilbert 
method and independently by the author in [15] using Wiener-Hopf and Toeplitz-Hankel 
operator techniques. Another proof was given by Deift, Its, Krasovsky, and Zhou [T2] also 
using the Riemann-Hilbert method. 

Another class of examples arises from the soft edge scaling of various random matrix 
ensembles. Such scaling leads to the Airy kernel [3B] • The corresponding asymptotics in the 
unitary case was proved by Deift, Its, and Krasovsky [TT], and another proof also including 
the orthogonal and symplectic case was provided by Baik, Buckingham, and DiFranco [8]. 

In order to give an outline of the paper and to make connections with other results, we 
need to introduce some notation. The Fourier transform J-" on L^(M) and its inverse will be 
written as 



The operators of primary interest to us are Wiener-Hopf and Hankel operators acting on 
They are defined as 



Here Mr (a) : / t— )■ a/ is the multiplication operator on L^(M) by a function a G L°°(]R), 
n_|_ stands for the orthogonal projection of L^(M) onto the subspace L^(R+), and is the 
flip operator {J^f){x) = f{—x). Basic information about Wiener-Hopf and Hankel integral 
operators can be found in [TOl Ch. 9], but we mention that the notation is slightly different 
there. 




(9) 



W{a) 



H+^MK(a)^-iH+U2(M^), 
H+ J'Mk (a) J- VkH+ I i2 ) . 



(10) 
(11) 
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In addition to the finite section projection Pr already defined in (jl]) we define the com- 
plementary projection = I — Pr acting on L^(R_(_). Finally, for complex parameters (3 
we introduce the following two functions defined on M, 

f ix + „ f 1 + ix\'^ 

= [j^;r^ ) ' «/3,oo(x) = ( j—^ 1 , xeR. (12) 

Here the principal values of the power functions are considered. The function U/3fi{x) is 
continuous on the one-point compactification M of M except at x = 0, where it has a jump 
discontinuity. The function Ui3,oo{x) is continuous on M and has (in general, different) limits 
at a; = ±oo. 



Let us now outline the proof of the asymptotic formula ([5]). It is split into several steps. 

Step 1: Here we will establish the following identity for each (fixed) R > under the 
assumption |Rea| < 1, 



p(")(i?)=exp(-^ + «i?)det[(p^(/ + //M(^))"'P/?) '[PRil + H^m-'Pi,)] (13) 
where 

(see Cor. 16. 3p . The expression under the determinant is of the form identity plus trace 
class operator on L^[0,i?], and thus the determinant can be understood as an operator 
determinant |20l[35]. The several inverses appearing in this expression are those of bounded 
linear operators acting on L^(M+) or L'^[0,R], resp., whose existence will be shown. 

The proof of identity f[T^ is carried out again in several steps (Sections miH]). The main 
idea is to discretize the Bessel kernel by the Jacobi kernel (Sec. In terms of random 
matrices, this corresponds precisely to taking the hard edge scaling limit of the JUE. In this 
way, one obtains an interesting limit relation involving Hankel determinants. 



/ d2 \ det ( 
pH(P) = exp ( + aR] lim ^ 

V 4 y n->oo / 



det [ J^-^ x^^^Wn{x) dx 



n-l 
j,k=0 



det ( J^-^ x^^''w{x) dx 



n-l ' 
j,k=0 



where i? > is fixed, and Wn and w are certain weight functions on [—1, 1] depending also 
on R and a (see Prop. 15.21) . Remarkably, this identity exhibits already the leading order 
terms of the asymptotics of P^°'\R) as R oo. 

Earlier, in Sec. H] we will establish a formula for Hankel determinants, which expresses 
them in terms of the discrete version of the types of operators appearing in (fT3l) . Such 
formulas have been established before in special cases [151 IE]- 
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In Sec. O using these formulas, we then take the hmit n ^ oo while keeping R > 
fixed, in order to arrive at f lT3|) . We remark that the weight in the Hankel determinants 
has singularities at —1 and 1, which is in a way the reason why we arrive at the rather 
complicated expression f lT5]) . 

Step 2: In this step we now focus on the expression 



One is tempted to write it as a quotient of two determinant. However, this is not possible 
because the underlying expressions are no longer of the form identity plus trace class. The 
reason is that both symbols if) and if) are functions with a jump discontinuity at infinity. 
However, the "size" of the jumps is the same for ip and if) so that in the expression (|T^ some 
kind of cancellation occurs. 

What we will show is that f|T^ equals asjTiiptotically [R — t- oo) 



This result will be stated only the final section (Sec. |8]), but the theorem leading to it is 
Thm. 17.61 which is proved in Sec. [3 The factor in front of the above quotient can be 
interpreted as some correlation with the part Ui/2+a,oo in the original symbols ip and 
which has been cancelled. 

Step 3: Here we apply a result, established by E.L. Basor and the author which 
computes the asymptotics of the determinants 



as -R — )■ oo. This quantity equals (up to a simple factor) a Wiener-Hopf-Hankel determinant 
with a specific Fisher-Hartwig type symbol (see ([T7|l below). Notice that it is the asymptotics 
of this determinant which contributes the Barnes G-function appearing in formula ([5]). With 
this last step the proof is complete (Thm. 18. 2p . 

In Step 1 and Step 2 we need two types of auxiliary results, which will be established in 
Sec. 2 and 3. One is the invertibility of certain operators, and the other one is the stability 
(i.e., asymptotic invertibility) of certain (generalized) sequences of operators. These results 
will be applied at several different places, and they are certainly of interest in their own 
rights. For this reason, we found it more suitable to establish these auxiliary results in two 
separate sections. Some notation and other basic auxiliary facts will also be stated in Sec. 2. 

Let us now make a connection with earlier results. In \^ it has been shown that 




(14) 






(15) 
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P(i/2)(i?) = exp(^-^ + ^)det^Pn{I-H^{ui/2,o))-'PR)- (16) 

It is not too hard to see that both these identities are special cases of (fT3l) . 

The types of determinants appearing in (fT5|) and (fT6|) . even for more general parameters, 
can be identified up to a simple factor with Wiener-Hopf-Hankel determinants [5]: 

det [PR{W{vy2+a) + H^ivy2+a))PR) = C-^^^/^+a) (^p^^j ^ i7j,(^_,/2_„,o))"'Pfl) (17) 

if |Rea| < —1, and 

det (P/j(W^(t)-l/2+a) - H^{v^y2+a))PR) = e-^(-l/2+") det (Ph(/ - H^{Ul/2-a,o)r'PR) 

(18) 

if < Re a < 1. Therein, Vy{x) = {x^/{l + x"^))^. Notice that the right hand side of ffTSl) 
makes sense for |Rea;| < 1. 

There is another way of looking at the determinant in the case of real a > — 1. In 
fact, the operator I — can be considered as a Bessel convolution operator with highly 
degenerate symbol. For a > —1, the (unitary and self-adjoint) Hankel transform is 
defined by 

POD 

H„ : L2(M+) i^^(M+), f{x) ^ g{x) = / Vt^Ja{tx)f{t) dt. 

Jo 

For a G L°°(]R+) we define the Bessel convolution operator Ba{a) as 

5„(a) = H,MK4a)H„ 

where M^^{a) stands for the multiplication operator on L^(M+) with symbol a. If a G 
Li(M+) n Li(M+), then B^{a) is an integral operator on L^(]R+) with the kernel 

/•oo 

Ba{a){x,y) = / ty/xyJa{tx)Ja{ty)a{t)dt. (19) 
Jo 

The Bessel operator B^ is a Bessel convolution operator where the symbol is the char- 
acteristic function of the interval [0,1], i.e., B^ = Pa(x[o,i])- Indeed, the kernel ([1]) can be 
rewritten as ^ 

Bo,{x,y) = ty/xyJa{tx)Ja{ty)dt, (20) 
Jo 

see, e.g., formula (25) in The quantities P'^")(P) are thus the determinants of finite 
sections of the Bessel convolution operators with the symbol 1 — xio,i], i-e., 

P(")(^) = ciet(P^P,(l - X[o,i])PR)\mo,R]. 
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This symbol is highly singular in the sense that in vanishes on a whole interval. It is 
thus worse than Fisher-Hartwig type symbols and leads to a different asymptotics. This 
may be a reason why the asymptotics of the determinants is difficult to compute. The 
determinants of Bessel determinants with smooth and regular symbols have been computed 
in [1], where the analogue of the Achiezer-Kac formula [I0| was derived. Determinants of 
Bessel operators with Fisher-Hartwig type symbols have not yet been investigated except for 
the cases a = ±1/2 (see [3 [7]). 

2 Notation and invertibility results 
2.1 Basic notation 

For 1 < p < oo let L^(T) stand for the Lebesgue spaces on the unit circle T = {2; G C : 
\z\ = 1}. By HP{T) and Hp{T) we denote the corresponding Hardy spaces, i.e., 

HP{T) = |/gLP(T) : /„ = for all n < }, (21) 

HP{T) = |/gLP(T) : /„ = for all n > |. (22) 

Here /„ denotes the n-th Fourier coefficient of /, i.e, 

/n = ^^'^/(e^^)e-^"^da;. 

Let us define the discrete analogues of the operators W{a) and H^{a). For a G L°°(T) 
the Toeplitz and Hankel operators acting on if^(T) are defined by 

T(a) = PrM{a)PT\H^T), H{a) = PjM{a)JPT\HHr), (23) 

where Pj is the orthogonal projection of L^(T) onto the subspace H'^(T) (i.e., the Riesz 
projection), Jt is the flip operator {Jff)(t) = t^^f(t~^), t G T, and M(a) : / a/ is the 
multiplication operator on L^(T). The following basic relations hold for a, 6 G L°°(T), 

T{ab) = T{a)T{b) + H{a)H{b), (24) 

H{ab) = T{a)H{b) + H{a)T{b). (25) 

Here and in what follows 

b{t) := b{t-^), t G T. (26) 

As special cases we obtain 

T{abc) = T{a)T{b)T{c), H{abc) = T{a)H{b)T{c) (27) 
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for a G iy°°(T), h G L°°(T), and c G if°°(T). For more information about Toeplitz and 
Hankel operators we refer to Ch. 2]. 

We are going to consider Toeplitz and Hankel operators with particular symbols, which 
involve the functions 

«^_^(e*V) = e^^(^-"\ 0<^<27r, r G T, /3 G C, (28) 

or, equivalently, 

Ufi,r{t) = (-V^)^ t e T (29) 

with the principle value of the power function considered. These function are nonzero and 
continuous on T \ {r} and have a jump discontinuity at t = r with one-sided limits equal to 
U/3,r(r ± 0) = e^^^^. 

The finite section projections acting on H'^(T) are defined by 

oo n—l 

Pn : 5^ /.e^'^ ^ 5^ ^e^'^ (30) 

fc=0 fc=0 

and we put Qn = I — Pn- There is a notational ambiguity with respect to the projections 
Pr and Qr acting on L^(]R_|_), but the context will make clear what is meant. 

The relationship between the discrete and the continuous case (except for the finite 
sections) becomes clear when introducing the map S : L^(T) i— )■ L^(R) defined as the com- 
position S = oU, where J-" is the Fourier transform and U is the unitary operator 



U:fe L\T) ^ge L\R), g{x) = — / 



1 + ix 



A/7r(l — ix) \1 — ix 

The restriction of S onto iJ^(T) is a mapping whose range can be identified with L^(]R+). 
We will denote this restriction also by S. It is straightforward to show (see also [10, Ch. 9]) 
that 

ST{a)S-^ = W{a), SH{a)S-^ = H^ia) with a{x) = a ( i±i^ ) . (31) 



1 — ix ^ 

The transformation a H- a also relates the specific symbols (fT2l) and ( !29l) to each other, 

We remark that up to a constant (due to our definition of the Fourier transform) the operator 
S is unitary. 
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2.2 Invertibility of operators / + H('iIj) and / + H^(il)) 

The goal of this section is to estabhsh sufficient invertibihty criteria for operators / + H{ip) 
and I + H]s.{i') acting on H'^(T) and L^(]R_|_), respectively, for particular piecewise continious 
symbols. 

Let W stand for the Wiener algebra on T, i.e., for the set of all a G L°°{T) such that 

oo 

ll'^llw '■= kn| < OO, 

n=— oo 

where the Fourier coefficients of a. Define 



w+ = wnH^{T), = wniJ°°(T). 

The sets W and yV± are Banach algebras with unit element. For an arbitrary Banach algebra 
B with unit element we denote by GB the group of invertible elements in B. 

The following result about the invertibility of / + H{iIj) generalizes previous results es- 
tablished in Sec. 3.2 of [5], Sec. 4.1 of [15j, and Thm. 3.2 of [16j. The present result can be 
generalized further, but the proof is more complicated [6j. 

In the proof given below we will use the notions of Fredholm operators and essential 
spectrum [20j. Recall that a bounded linear operator A acting on a space Hilbert space H 
is called Fredholm if it has a closed range and if its kernel, ker A, and its cokernel, H/imA, 
are both finite dimensional. The Fredholm index is the difference between the dimensions 
of the kernel and cokernel. The essential spectrum of A is the set of all complex numbers A 
such that A — XI is not a Fredholm operator. 

Theorem 2.1 Let a, f3 E C be such that \Rea\ < 1 and |Re/3| < 1, and let c+ G GW+. 
Then the operator 

I + H{tp) with = C+C^^U^i/2-a,lUi/2+l3-l, 

is invertible on H'^{T). 

Proof. We first use a result of Power [501 EI] (see also [101 Sec. 4.7]) in order to show 
that the operator / + H{ip) is a Fredholm operator. The result of Power states that the 
essential spectrum of a Hankel operator H{ip) with piecewise continuous symbol ip equals 
the following union of closed intervals in the complex plane, 

spessH{^) = [0,iij-i]U[0,-ii!i]U y ^-i^/4^,i^/4^ , 

Im {t)>0 
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where 

= -(^(r + O) -V(t-O)) with ij{T ±0) ^ lim ^{re''). 

In our case we have ipT — ior r e T, Im (r) > 0, and 

ipi = icos(Q!7r), — — icos(/37r). 

Hence the essential spectrum of H{t/j) is [0, cos(Q;7r)]U[0, cos(;57r)]. The point A = — 1 does not 
belong to the essential spectrum if and only if Re a and Re f3 do not belong to the set 1 + 2Z. 
This is fulfilled in our case. Hence we conclude that I + H{il)) is Fredholm. Moreover, making 
use of the fact that the complement of the essential spectrum is connected and the fact that 
the Fredholm index is invariant under small perturbations it follows that the Predholm index 
of / + H{'^) is zero. 

It remains to show that the kernel of / + H{'^) is trivial. For r e T, we introduce the 
functions 

r^,At) = (1 - t/r)\ i,,r{t) = (1 - r/tf, teT\ {r}, 
where the principal values of the power functions are considered. We notice that 

^7,r(i) = i-iA^) • V-yAt) and r7^,r(^"^) = ^7,r-i(^)- 

Now assume that /+ € H'^(T) belongs to the kernel of / + H{ijj). Then there exists /_ e 
lP(f) such that 

U{t)+mt-'f+{t)=t-'f4t). 

We rewrite this as 

tf4t)+mu{t) = f-{t): 

and decompose (using = —t) 

in order to obtain 

Because of the assumptions on a and /3, the function c]^^ii2-a,iii/2+p-i belongs to H'^iT). 
Hence the right hand side of the above equation belongs to if^(T). Each of the terms on the 
left hand side belongs to L^(T) and /o = — /o- Comparing the Fourier coefficients it follows 
that /o = 0, whence f- — and 
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Now we decompose (using U-i-i(t) = t ^) 
in order to obtain 

The function belongs to H^{T), and the last equation states that 5^+ = —g+. It follows 
that g+ = 0, and from this that /+ = 0. Hence the kernel of / + H{ip) is trivial. Since we 
have already shown that / + Hlip) is a Fredholm operator with index zero, this implies the 
invertibility. □ 

The analogue of the previous result in the continuous case is stated next. For simplicity, 
we restrict ourselves to symbols in which c+ = 1, which is enough for our purposes. 

Corollary 2.2 Let a,/? G C be such that |Rea;| < 1 and |Re/3| < 1. Then the operator 

I + HK{U-l/2-afiUl/2+/3,oo) 

is invertible in L^(]R+). 

Proof. We apply the transform S : if^(T) — > L^(]R_|_) to the Hankel operator H[iIj) and 
obtain 

by using the formulas (l3T!l and (1321) . Now the result follows from the previous theorem. □ 

3 Several results about stability 
3.1 Definitions and basic results 

We need the notions of stability and strong convergence. These notions involve generalized 
sequences of operators. Let A C M be an index set and assume that the supremum Aqo = 
sup A G M U {+c>o} does not belong to A. For us, only the following three settings are of 
interest, 

(i) A = and Aqo = +00, 

(ii) A = (0, 00) and Aoo = +00, 

(iii) A = [0, 1) and Aoo = 1- 



13 



We consider generalized sequences of bounded linear operators {Ax}x^\, where Ax E 
C{Hx) and Hx are Hilbert spaces. Such a sequence is called stable if there exists a Aq G A 
such that the operators Ax are invertible for all A > Aq, A G A, and if the inverses are 
(uniformly) bounded, i.e., 

sup \\Al^\\c(H^) < oo. 

A>Ao 

Stability is also referred to as asymptotic invertibility. 

Given a sequence {AaIasa, let us assume that the spaces Hx are subspaces of a possibly 
larger Hilbert space H , and let A G C{H). We say that Ax ^ A converges strongly on H as 
A Aoo if for each x G if we have 

lim \\AxPxx — Ax\\H = ^- 

Therein, Px stands for the orthogonal projection from H onto Hx- 

We will always be concerned with bounded sequences, i.e., sequences {Ax\xeK for which 

sup Pa||£(//;,) < oo. 
AeA 

Such a sequence is called a zero sequence if 

lim WAxWciH^) = 0. 

The above definitions and the following basic results can be found, e.g., in (TUl Sec. 7.1]. The 
adjoint of a linear bounded operator A acting on a Hilbert space will be denoted by A*. 

Lemma 3.1 Let {Ax} be a bounded sequence of operators Ax G C{Hx)- Then the following 
holds: 

(i) {Ax} is stable if and only if there exists a bounded sequence {Bx} such that {AxBx — 1} 
and {BxAx — 1} are zero sequences. 

(a) If {Ax} is stable, and Ax ^ A and — t- A* strongly, then A is invertible. 

(Hi) If {Ax} is stable, A is invertible, and Ax ^ A strongly, then A^^ — t- A^^ strongly. 

The significance of the previous lemma is the following. Statement (i) implies that 
stability is invariant under perturbation by zero sequences. It also justifies to refer to stability 
as asymptotic invertibility. Statement (iii) indicates the reason why we are interested in 
stability. In fact, we frequently need to show the strong convergence of a sequence of the 
inverses of operators. Finally, statement (ii) shows that in the cases of interest to us (in 
which we will always have strong convergence Ax ^ A and — )■ A*) the assumption in 
(iii) that A be invertible is also necessary. 
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Since we will be dealing with operator determinants we need the notion of trace class 
operators [20] [35]. Recall that a compact operator A acting on a Hilbert space is called trace 
class if the sequence of its singular values (i.e., the eigenvalues of (A*AY^'^) is absolutely 
summable. The sum of the singular values is by definition the trace norm of A. 

We often want to show convergence in the trace norm. Here the following basic statement 
is the key (see also [lOj Sec. 1.3]). 

Lemma 3.2 Assume that K is a trace class operator and that Ax ^ A and B* 
strongly. Then A\KBx — ^ AKB in the trace norm. 

In order to show the strong convergence of certain operator sequences we will frequently 
employ the following result (see, e.g., [T71 Lemma 5.4]). Therein, the measure is meant to 
be the Lebesgue measure. 

Lemma 3.3 Let K = or K = T. Let {&a}aga be a hounded sequence of functions in 
L°°{K), and assume that bx converges in measure to b E L°°{K). Then 

W{hx) ^ W{h), H^{hx) ^ H^{h), W{hxy ^ W{h)\ H^ibxT ^ H^{b)\ 

strongly on L^(]R+) in the case K = M., and, 

Tibx) ^ r(6), H{bx) ^ Hib), TibxT -> Tib)*, Hibx)* ^ H{b)\ 

strongly on iJ^(T) in the case K = T, respectively. 

In the following two subsections we are going to use the following auxiliary result as well 
(see [ini Prop. 7.15]). 

Lemma 3.4 Let A he an invertihle hounded linear operator on a Hilhert space H . Let P he 
a hounded projection operator on H , and let Q = I — P . Then PAP is invertihle on imP if 
and only if QA^^Q is invertihle on imQ. In particular, 

(PAP)-^ = PA-^P - PA-^Q{QA-^Q)-^QA-^P, 
iQA-'gy' = QAQ-QAP{PAP)-'PAQ. 

To be more specific, in the previous lemmas the operators PAP and QA~^Q are considered 
as the restrictions onto the spaces imP and imQ, respectively. 
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3.2 Stability of certain finite sections (continuous case) 

The goal of this subsection is to prove the stabihty of the following sequence of finite sections, 

{PR{I + H^{^))-'PR}RyO, (33) 

for certain symbols ip G PC(M). Here PC(M) stands for the set of all piecewise continuous 
functions on the real line (with limits as x — i- ±00). 

We remark that the main result of this subsection (Corollary 13. 9p could also be derived 
from results of Roch, Santos, and Silbermann [33]. However, some additional effort would 
still be necessary in order to identify certain operators which are defined by homomorphisms 
and to show the invertibility of these operators. In fact, these invertibility results would 
correspond to Corollary 12.21 and Proposition 13.51 below. We found it easier to give a direct 
proof in the special case that is of interest to us. Nonetheless, Corollary 13.91 can probably 
be generalized to a larger class of symbols, in which case a direct application of the results 
[33] is perhaps more suitable. 

Introduce the function 

,...,.( i if X > 
X{x) = I ■ sign(x) = { _^ if X < 

In a first step we establish an invertibility or, more precisely, spectral result. 

Proposition 3.5 For R > 0, the operators QrH-^{x)Qr\l^{r,oo) o'^e self-adjoint, unitarily 
equivalent to each other, and have spectrum equal to the interval [0, 1]. 

Proof. Let us consider the Fourier convolution operator 

Wo (a) = TMR{a)T~\ 

where we use the same notation as in f|TT]) . Then the operators QrHk{x)Qr are equal to 
the compressions of the operator Wo{x)Jr oiito L^(i?, 00). It is well known (see, e.g., [2^) 
that the Fourier convolution operator W^o(x) is equal to iS, where 5* is the singular integral 
operator on M, 

{Sf)ix) = - r ^dy, fEL'{R). 
J X-y 

Therein the integral exists a.e. as the Cauchy principle value. It follows that the operator 
Hr{x) is an integral operator on L^(M+) with the integral kernel tt~^{x + y)"^. 

On the other hand, H-r[x) is a Mellin convolution operator Mo(n) with the generating 
function 

n{z) = (cosh(7r2))"\ z eR. 
16 



To see this, recall that a Mellin convolution operator is defined as Mo(a) = Ai ^MK{a)Ai, 
where 

poo 

M : L\R+) ^ L2(M), (-M/)(0 = / fix)x'^'^/^dx 

Jo 

is the Mellin transform. Obviously, Ai = J-^^T, where T is the isometry defined by 

r : L2(M+) ^ L\m), {Tf){x) = f{e^)e^l\ 

Therefore the identity Hr{x) = Mo{n) reduces to 

TH^{X)T^' = 7M^{a)F-^ = WQ{n). 

This identity can be verified directly by showing that the Fourier transform of n{z) is 
(27rcosh(x/2))~-^ and by observing that TH^{x)T~^ is an integral operator on L^(M) with 
the integral kernel equal to (27rcosh((x — y)/2))^^. 

Finally, observe that the transform A ^ TAT^^ maps the operator Qr into MTgi{x{\nR,oo)) ^ 
Hence the operators Q rH^{x)Q R ai'e unitarily equivalent to 

{X{\nR,oo))WQ{n) Mm (X(ln R,oo))\ (In i?,oo) , 

which (by means of a translation) are unitarily equivalent to the Wiener-Hopf operator W{n). 
In particular, they are unitarily equivalent to each other. Because the symbol n{z) is real- 
valued, continuous on the one-point compactification M, and has range [0, 1], it follows from 
basic Wiener-Hopf theory that the operator W{n) is self-adjoint and has spectrum equal to 
[0,1]. □ 

The fact that the operators QrH^Xx)Qr\l'^(r,oo) are unitarily equivalent to each other 
can also be seen in a more direct way. Introducing the unitary operators Y/j : /(x) i— t- 
i?~^/^/(x/-R), acting on L^(M+), one can show easily that 

QiHMQi = Yr^QrHMQrYr- (35) 
Moreover, we can immediately conclude the following result. 

Corollary 3.6 Let a E C be such that |Rea| < 1. Then the operators 

Pr{I + Ht>{U-l/2^a,0Ui/2+a,oo)y^PR\L^[0,R] (36) 

are invertible for each R> and the inverses are uniformly bounded. 

Proof. The invertibility of / + H^{u^i/2-a,oUi/2+a,oo) follows from Corollary 12.21 Hence, 
using Lemma 13. 4[ the invertibility of the above operators is equivalent to the invertibility of 

QRil + Hf>{u_i/2-a,0Ul/2+a,oo))Q R- 
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Now observe that 

... ^ , j e^"(i/2+a) if a; > 

U-l/2-afi[X)Ui/2+a,oo[X) — < g_ivr{l/2+a) [{ x < 

which is a constant function plus the function cos(7ra)x(x). The last operator thus equals 

Qr + cos{7ra)QRHs.{x)QR, 

which is invertible by the previous proposition if and only if cos(7rQ;) ^ (— oo,— 1], i.e., 
Re a ^ 1 + 2Z. Since for different R these operators are unitarily equivalent to each other, 
the uniform boundedness of the inverses follows immediately. Applying the formulas for the 
inverses stated in Lemma [3.41 it follows that the inverses of f p6|) are uniformly bounded, too. 

□ 

Let M stand for the two-point compactification of M, and let C(R) stand for the set of 
all continuous functions on M. 



Lemma 3.7 Let a G C(R). Then \\QRHK{a)Qii\\c(L^{R,oo)) — > as R oo. 

Proof. A Hankel operator //^(a) with a G C(M) is compact. Because Qr — )■ strongly as 
i? — )■ oo, the statement is proved for such symbols. To extend the result to arbitrary functions 
in C(R) it suffices to consider the symbol a{x) = x{^) ~ which is continuous on R, 

but has different limits as x — )■ ±oo. As shown in the proof of the Proposition 13. 5[ H]^{x) is 
an integral operator on L^(R+) with integral kernel n'^lx + y)~^. By computing the Fourier 
transform of xe~'^' we see that Hm.{xg~^^^) has the integral kernel 7r~"^(l + x + y)~^. Hence 
H^{a) has the kernel {n{x + y){l + x + y))~^- It is now easily seen that QRH^{a)QR\L2(^R^^^ 
are Hilbert-Schmidt operators with the Hilbert-Schmidt norm converging to zero as i? — )■ oo. 
This implies the desired assertion. □ 

The key point to the treatment of the stability of f p3|) is the following result. Therein the 
generating functions are allowed to have jump discontinuities at x = and x = oo. Notice 
that the size of the jump at x = oo does not play a role. 

Theorem 3.8 Let a G PC(R) be continuous on R \ {0}. Then the generalized sequence 
{Qr + Qij-f^R(a)(5ij}^>o is stable if and only if 

^:^'''+»>-°'-°'^(-oo.-l]. 

Proof. Given a we write 

a(x) = (3 x(x) + 6(x) 
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with b G C(]R). Because of the previous lemma, the sequence QRH^{h)QR converges to zero. 
Hence the sequence under consideration is stable if and only if so is Qr + (iQRH-^{x)QR 
(see Lemma [3.1( i)). However, the necessary and sufficient condition for the stability of this 
sequence was already identified in the proof of Corollary 13.61 by help of Proposition 13.51 □ 

The desired stability result is the following. We restrict ourselves to the cases which are 
of interest to us. 



Corollary 3.9 Let a ^ C be such that |Rea| < 1. Then the generalized sequences 
are stable. 

Proof. By Corollary 12. 2[ the assumption on a implies that the operators / + H^{u~i/2~afi) 
and I + H^{ui/2+a,oo) are invertible on L^(R+). Hence we can apply Lemma [3^ (including 
the formulas for the inversess) and reduce the stability statements to the stability of 

{Qr{I + HR{u_i/2-a,o))QR}^^Q and {Qr{I + HK{ui/2+a,oc))QR} j^^q , 

respectively. By Lemma [3771 the second sequence is always stable, regardless of the parameter 
a. For the first sequence we can apply the previous theorem, and therefore we compute the 
parameter 

/3 = = = cos(7ra). 

The condition cos(a7r) ^ (—00, —1], i.e., Rea ^ 1 + 2Z, is satisfied by assumption. □ 



3.3 Stability of certain finite sections (discrete case) 

Now we are going to present the discrete versions of the previous results and prove the 
stability of sequences 

{PM + H{^)y'Pn}n&^ (37) 

for certain symbols G PC{T). Here PC{T) stands for the set of all piecewise continuous 
functions on T. 

Also in this case there exists a general stability criterion, which was established by Roch 
[32] and from which the main result of this subsection (Cor. I3.12p could be derived. As 
before additional effort would still be necessary. Therefore we found it easier to give a direct 
proof for the particular cases we are interested in. A generalization of Corollary 13. 121 to more 
general symbols is certainly possible. 
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Proposition 3.10 Let a G PC{T) be continuous on T \ {1}. Then the sequence {Q„ + 
QnH{a)Qn}nez+ Stable if and only if 

,3:^°'^+°'-''''-°'^(-oo.-l|. 

Proof. Let a{e^^) = i{7c — x)/tt, < x < 27t. Then we can write 

a{e'^) = /3(T(e"') + 6(e") 

with b G C(T). Because H{b) is compact and Qn strongly, the stability of Qn + 
QnH{a)Qn is equivalent to the stability of Qn + f3QnH{(r)Qn (see Lemma IXlT i)). 
Let us introduce the operators 

oo oo 

which are isometrics. Their adjoints are given by 

p{k+l)/n 

K:L2(R+)^ff2^T), with fk = V^ f{x)dx. 

u—n J kin 



k=0 



Now consider the operator ElH^{x)Ei. In the matrix representation of this operator 
with respect to the standard basis {t"}5^Q of H'^(T), the (j, fc)-entry equals 



- / -^dxdy= +0{{j + k + iy'), j,k>0. 

TTjj Jk x + y 71(1+ ] + k) 



The error term corresponds to a Hilbert-Schmidt operator K. Because the n-th Fourier 
coeffcient of a evaluates to (vm)^^, n 7^ 0, we obtain from the definition of the Hankel 
operators that ElH^{x)Ei = H{a) + K. From this we now conclude that the stability of 
Qn + /3QnH{(T)Qn is equivalent to the stability of Qn + l3QnElH^{x)EiQn (see Lemma [3?]T i) 
and use the fact that QnK — > as n — )• 00). 
Now we observe that 

Q„ElH^{x)EiQn = QnE*H^{x)EnQn = E*QiHR{x)QiEn- 

Here we have used the invariance of H^{x) under contraction. Hence we arrive at the 
sequence 

E:{Q, + (3QiHu{x)Qi)En. (38) 

Because En is an isometry, it follows from Prop. 13.51 that E*QiH^{x)QiEn is self-adjoint 
and has spectrum contained in [0, 1]. Thus, ii /3 ^ (—00, —1], then the sequence is stable. 
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Conversely, if the sequence (l38l) is stable, then so is 

e^e:{q, + pq,h^{x)Qi)e^e:. 

Now observe that EnE* converges strongly to the identity operator on L^(R+) as n — oo. It 
follows that Qi + f3QiH^{x)Qi is invertible, which by Prop. 13.51 implies that f3 ^ (—00, — 1]. 

□ 

Theorem 3.11 Let a G PC be continuous on T \ {1,-1}. Then the sequence {Q„ + 
QnH{a)Qn}n&+ is stable if 

a(l + 0)-a(l-0) , , , a(-l-0)-a(-l + 0) , , 

^ ^^-^ i (-00, -1] and ^^-^ i (-00, -1]. 

Proof. Let us write a = 01 + 02, where ai is continuous on T \ {1} and 02 is continuous 
on T \ { — 1}. The previous proposition implies that the sequences Qn + QnH{ai)Qn and 
Qn + QriH{a^)Qn are stable where a^{t) = —a2{—t). By applying the flip f(t) f{~t) 
(which is an isomorphism) to the left and the right hand side of the last sequence it follows 
that Qn + QnH{a2)Qn is stable. 

Our next claim is that QnH{ai)QnH{a2)Qn tends to zero in the operator norm as n — t- 00. 
To see this, let 14„ = T(t±"). From ([27]) we have Q„ = K^-n and V-nH{h) = H{bt-'') = 
H[b)Vn- Hence we can write 

QnH{ai)QnH{a2)Qn = QnV^nH{ai)H{a2)VnQn- 

Notice that V^n ~^ strongly and that the product H{ai)H{a2) is a compact operator. The 
latter holds because we can write 

H{a{)H{a2) = H{a{)T{h)H{a2) + H{a{)T{f2)H{a2) 

= (i/(ai/i) - T(ai)/J(/i))if(a2) + H{a^) (1/(^2) - H{f2)T{h2)) 

with continuous even functions /i,/2 satisfying /i + /2 = 1 and such that fk is identi- 
cally zero on a neighborhood of the discontinuity of Ofc. A similar argument shows that 
QnH{a2)QnH{ai)Qn converges to zero in the operator norm. 

Now abbreviate An = QnH{ai)Qn and i?„, = QnH{a2)Qn- Then 

Qn + QnH{a)Qn = Qn + + 5„. 

We claim that Qn — {Qn + An)~^An — {Qn + Bn)^^Bn is an asymptotic inverse. If we multiply 
these two expressions we obtain 

Qn - {Qn + AnY^An - {Qn + Bn)'^ Bn + A„ - (Q„ + AnY^ AI + 5„ - (Q„ + Bn^^Bl 
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plus terms tending to zero in the norm (because they contain the products AnBn or BnAn). 
The last expression simplifies to Qn- This completes the proof. □ 

One can show that the previous result is actually an "if and only if" statement. Since 
we will not need it we omit a proof. Our desired result of this section is the following. 

Corollary 3.12 Let G C he such that |Rea| < 1, |Re/3| < 1, and let c+ G GW+. Then 
the sequence 

[Pn{I + H{'4)))~^Pn]^^^^ with tp = C+C;^M_i/2-a,lUl/2+/3,-l 

is stable. 

Proof. The invertibility of I + H{ip) on iJ^(T) follows from Theorem 12.11 Using Lemma [3l4l 
the stability can be reduced to the stability of Qn{I + H{ip))Qn. We can apply the previous 
theorem, and for this we compute 

+ 0) - - 0) e^^(l/2+a) _ g-^i(l/2+a) 

— = cosivra), 



2i 2i 
_ 0) - ^(-1 + 0) e"-'(i/2+/3) _ e--{i/2+/3) 



cos(7r/3). 



2i 2i 
The corresponding conditions are fulfilled if and only if Re a ^ 1 + 2Z and Re /3 ^ 1 + 2Z. □ 



3.4 Stability results for operators with approximating symbols 

In Sections H] and [7] on we are going to approximate operators of the form 

with a certain symbol (p ^ PC{T) by operators of the same type but with smooth symbols 
(j)^. Clearly, we cannot expect approximation in the norm. What is sufficient for our purposes 
is the approximation in the strong operator topology. In view of Lemma I3.1( iii) we need 
to examine the stability of operators / + where /i G [0,1) is the approximation 

parameter. The stability result is non-trivial, and we make in fact use of results established 
by the author and Silbermann in |17j . 

We will use the results of this subsection at two different places, namely in the proof of 
Theorem 14.21 and Proposition I7.5[ 

Let us introduce the necessary notation. For fi G [0, 1) and r G T, define the composition 
operators and Yr acting on L°°{T) by 

(Gjm = f [l^) > (Yrfm = /(rt), (39) 
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and the operators i?^ acting on L^(T) by 



wm = I . (40) 



1 + fit \1 + fit 

The operator Y^- also acts on L^(T), and both and Y^- are unitary operators. The Hardy 
space if^(T) is an invariant subspace of i?^ and Y^ as well as of their adjoints. We will use 
the same notation for the restriction of i?^ and Y^- onto if^(T). These restrictions are unitary 
operator on if^(T), too. Moreover, for G L°°{T) we have 

R,H{ct>)R; = HiG,^), R,T{<P)R; = T{G,ct>). /i e [0, 1), (41) 

and 

YrH{(P)Y; = TH{Yr(p) ifr = ±l, KT(0)F; = r(n0) ifreT. (42) 
The just mentioned statements are easy to prove (see also [T7| Sec. 5.1]). 
Theorem 3.13 Let a, /3 E C be such that \Rea\ < 1 and |Re/3| < 1, let c+ G GW+, and let 

Then the (generalized) sequence of operators 



is stable on H (T 



Proof. In order to prove the stability we apply the results of [T7t Sees. 4.1-4.2]. These 
results establish the existence of certain mappings $o and r G T, which are defined as 

^oii'f,] ■= /i- lim ipi„ $r[^M] := ^- lim G^Y^ipf,, 
where fi- lim stands for the limit in measure. It is easy to see that 

$oM = M-l/2-«,l, ^o[bp] = M1/2+/3 -1, 

and 
and 

^Ab^] = M1/2+/3 -1(7-), if r 7^ -1, ^-i[bp] = M-i/2-/3,-i- 
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Since = c+c^^a^h^ we conclude 

= C+C^^M_i/2-a,lMl/2+/3-l, 
= Ml/2+a,-l, 
^-lii^fi] = M-l/2-/3,-l, 

"^•'-[V'^t] = constant function, rGT\{ — 1,1}. 

The stabihty criterion in [17] (Thm. 4.2 and Thm. 4.3) says that {/ + i/('?/'^)}^jg[o,i) is stable 
if and only if the operators 

(i) ^o[/ + H{ij^)] = 1 + H{(^o[^^]) =1 + if(g+C;^M_i/2-a,l^^l/2+/3,-l), 

(ii) ^,[1 + Hiij,)] = 1 + H{<l>^[i,,]) = 1 + H{uy2+a,-l), 

(iii) + Hii,^)] = I - = / - if(w_i/2-/3,-l), 

(iv) ^Al + Hii^,)] = 

I 0\ f P \ ( Mi^^ilj^]) \fOl\fPO\^flO 

Oi)^[oq)[ M{^^,]) )[l )[0 Q J [O I 

(r e T, Im(r) > 0) 
are invertible. 

The operator / — H{u^i/2-i3-i) is invertible if and only if / + iJ('U_i/2_/3,i) is invertible 
because these two operators can be related to each other by a rotation operator F_i : f(t) ^-^ 
f{—t) acting on H^{T) (see fH2]) ). Now the invertibility of the operators (i)-(iii) follows from 
Theorem 12.11 □ 

4 A formula for Hankel determinants 

For b G L^[—l, 1], let i/„[6] stand for the n x n Hankel matrix 

Hn[b] = {b,+k+i)T,lo, (43) 

where 

1 

bk = - b(x)(2xf-^dx (44) 

J-l 

are the (scaled) moments of b. In this section we are going to derive a formula of the type 

det Hn[b] = G" det (p„(/ + H{ij))-^Pr^^ 
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where b G 1,1] is function of the form b{x) = (1 — x)°(l + x)^bQ{x) and b^lx) is a 
sufficiently smooth and nonvanishing function on [—1, 1]. The constant G and the function 
ip G PC{T) depend on the function b. This formula will allow us in the next section (Sec. [5]) 
to express the determinant of a Hankel matrix as a determinant of the type appearing on 
the right hand side. The invertibility of / + H{iIj) will be guaranteed by Theorem 12.11 The 
P„'s are the finite sections (l30ll . The above formula is a generalization of formulas of the 
same type established in [151 US] for particular values of a, (3. 

Recall the definition of the Wiener algebra W given in Sec. 12.21 A function a G W is said 
to admit a canonical Wiener-Hopf factorization in W if it can be represented in the form 

a(t) = a_(t)a+(t), t G T, (45) 

where a± G GW±. It is well known (see, e.g., [10]) that a G W admits a canonical Wiener- 
Hopf factorization in W if and only if a G GW and if the winding number of a is zero. This, 
in turn, is equivalent to the condition that a possesses a logarithm log a G W. In this case, 
one can define the geometric mean 

G[a] := exp(^^^ loga(e*^) d^) . (46) 

This definition does not depend on the particular choice of the logarithm. 

We will assume that a G GW is an even function. Then a has winding number zero and 
thus possesses a canonical Wiener-Hopf factorization. Moreover, the factors are related to 
each other by a_ = 7a_|_ with some nonzero constant 7. 

The following theorem is cited from [15l Thm. 4.5]. It is the immediate consequence of 
two other results, namely, Thm. 2.3 of [3] and Prop. 3.9 of To give some, but not all 
details, we remark that the last two mentioned results establish the identites 

deti/„[6] = det (p„(T(a) + H{a))P^ 

and 

det {Pn{T{a) + H{a))Pn) = G[a]"det (p„(J + H{i;))-^Pn). 

Therein it is necessary to assume that the symbols are smooth (or, more precisely, that 
the Wiener-Hopf factors of a are bounded). We remark that the invertibility of / + H{ip) 
is guaranteed again by Theorem 12. H which can be applied with the jump functions being 
absent. A more elementary argument for the invertibility of I + H{ip) with (certain) smooth 
symbols ip and an explicit formula for the inverse is given in Prop. 3.9 of 

Theorem 4.1 Let a G GW be an even function which possesses a Wiener-Hopf factorization 
a{t) = a-{t)a+{t) . Define ip{t) = a+{t)a'^^{t) and 

If n\ I ie\ / 1 + cos 61 , ^, 

Kco=9) = a(e-)^^— ^. (47) 
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Then I + H{iIj) is invertible on if^(T) and 

detK^lb] = G'[a]"det(p„(/ + if(^))-ip„). (48) 

The next theorem, which is the main result of this section, follows formally from the 
previous one with 

a{e'') = c{e''){2 - 2 cos ^)^/2+"(2 + cos^)-^/'^^ 
a+{t) = c+(t)(l - + t)-V2+/3^ a^it) = c_(t)(l - + t-i)-V2+/3 

if one considers the generalized Wiener-Hopf factorizations 

«-i/2-a,i(t) = (1 - tr'/'~-{i - 1-^'^-, = (1 + ty/'-^i + t-'r'/'^^. 

Of course, therein the Wiener-Hopf factors and the inverses are not bounded. In fact, in 
order to make the argument precise, we have to use an approximation argument, and we 
have to apply the stability results of Section 13.41 

Theorem 4.2 Letc G GW be an even function which possesses a Wiener-Hopf factorization 
c(t) = c_(t)c+(t). Define 

iPit) = C+{t)c-\t)u^i/2-a,l{t)Ui/2-pMt), teT, (49) 

b{cose) = c(e*^)(2-2cose)"(2 + 2cos^)'^, (50) 
and assume that \Rea\ < 1, |Re/3| < 1. Then 

det Hn[b] = Gfc]" det (p„{I + if(^))-^P„ 



Proof. For /i G [0, 1) we define the even functions 

/ \l/2+Q/ , \-l/2+/3 

«mW = c(t)((i-/it)(i-/ir')j [{i + fit){i + fit-')j , teT. 

By formula (H7j) . these functions give rise to functions 

, (l + /i^-2/icosg)V^+" (l+/.^ + 2/icosg)-V^+^ 
o^icosyj-c^e j- (2 - 2 cos^^)i/2 ' (2 + 2cos^)-V2 " 

Because 6^ — i- 6 in the norm of 1, 1] as /i — )■ 1, it follows that (for fixed n) 

H^[b] = ]imH^[b^]. 
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Furthermore, we observe that the canonical Wiener-Hopf factorization of is given by 

O'f.it) = a^^_^{t)a^^^{t) with 

a^,+ (t) = c+(t)(l-/it)i/2+"(l + /it)-i/2+^. 



Then 



Using the fact that G[a] = G[c\ and employing Theorem 14. II it follows that 

det H4b^] = G[c]"det (p„(/ + H{i;^))-^Pn 



whence 



det H^[h] = G[cY lim det f P„(/ + H{i,^))-^P^ . 



Because 



in measure as /i — )■ 1, it follows that ^ ip m measure as /i — )■ 1. Hence, by Lemma (3 .31 
H{ip^) ^ H{;il)) strongly on H^iJ) as /x ^ 1. 

Now Theorem 13.131 implies that the sequence {/ + if('?/'^)}^g[o,i) is stable, while Theorem 
12. II implies that the operator I + H{ip) is invertible on iJ^(T). Hence (see Lemma [3.1( iii)) 



strongly on iJ^(T) as /i — 1. This allows us to conclude that (for fixed n) the matrices P„(/+ 
H{ip^))~^Pn converge to P„(/ + if(-?/^))~^P„ as /i — )■ 1. Consequently, their determinants also 
converge. This completes the proof. □ 

We conjecture that the conditions on the parameters a and /3 in the previous theorem 
cannot be weakened. This is despite of the fact that the Hankel determinant H^^] is well 
defined if Re a > —1 and Re /3 > —1, and that the inverse of / + H{'il)) exists under the 
same condition and the extra condition that Re a ^ 1 + 2Z and Re /3 ^ 1 + 2Z. (For the last 
statement see |5l Sec. 3.2] or [6].) 
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5 The approximation of the Bessel kernel 

After having established the necessary auxihary results in the previous sections we are able 
to start with analyzing the quantity 

P(")(i?) = det(P^ - PBB^PR)yyo,RV 

As outlined in the introduction our first goal (Step 1) is to establish an alternative expression 
for this quantity, and this will be accomplished at the end of the next section. 

In this section we are going to approximate the Bessel kernel by the Jacobi kernel. This 
approximation corresponds to the fact that correlation functions for the Jacobi Unitary 
Ensemble yield in the hard edge scaling limit the Bessel kernel. 

It is possible to consider the Bessel kernel Ba{x, y) for complex parameters a with Re a > 
— 1, and the Jacobi kernel for complex parameters with Rea > —1, Re /3 > —1. Recall 
that the Jacobi kernel is defined by 

n-l 

Ki'''^\x,y) = ^w{x)w{y) ^Pk{x)pk{y), -l<x,y<l, (51) 

where w(x) = (1 — x)"(l + x)^ is the underlying weight, and pk{x) are the normalized 
orthogonal polynomials, i.e., 

1 

Pj{x)pk{x)w{x) dx = 6jk, (52) 

-1 

which are, of course, up to a constant equal to the Jacobi polynomials. We define the rescaled 
kernel as 

Ki-'^\x,y) = ^Kt'\l - ^, 1 - ^), < x,y < 2n. 

For sufficiently large n (namely, 2n > R) we can think of iri"'^^ as an integral operator 
acting on L'^[0, R], where i? > is considered fixed. 

From (IST!) it is clear that the operators K^C''^'' and KiC'^^ are finite rank operators with 
rank equal to n (or at most n). 

Theorem 5.1 Let R > be fixed. 

(i) For Re a > —1 the operator PRBaPR\L'2[o,R] is trace class, and hence the operator 
determinant 

P(")(P) = det(P^ - PRB^Pn)\L^[o,R] 

is well defined. Moreover, the mapping a i— PRBaPR\L'2[o,R] is an analytic (trace class) 
operator-valued function. 
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(ii) For a, (3 > —1 being real, we have PrKu Pr — )■ PjiBaPR in the trace norm and hence 

P^"\R) = lim det{PR - PRKt'^^Pn). (53) 

n— >oo 

Proof, (i): The Bessel operator is by definition the integral operator with the kernel 
([1]). Because this kernel can also be expressed by (120|) . we can write PrI^qPr = XR^^Xji^a 
where 

X^,, : L2[0, 1] ^ L2[0, R], Yr,^ : ^^[0, R] L^[0, 1] 
are integral operators with the kernels 

XR^aix,t) = VtxJaitx), YR^a{t,y) = VtyJaity). 

Using the asymptotics of the Bessel function at zero, Ja(t) ~ (V2)"/r(l + it is easy 
to conclude that both Xr^o and Yr^q are Hilbert-Schmidt operators (see [351 Thm. 2.11]). 
Hence PrBoiPr = XR^oXR,a is trace class. 

It is a basic fact that the function z'^Jaiz) =: g{a,z) is entire in both a and z (see 
[29| Sec. II, 9.3]). Thus, differentiating \/txJa{tx) = (tx)°'^^^'^g{a,tx) with respect to a 
yields \rL(tx)(tx)°'^^^'^g{a,tx) + (tx)°'^^^'^{dag){ci,tx). This implies that the derivative (with 
respect to complex a) of the operator-valued function a h- )■ Xr^o exists and is in fact a 
Hilbert-Schmidt. The same holds for the derivative of Yr o,- Hence PrB^Pr has a complex 
derivative with respect to a, which is trace class. 

(ii): This part was proved by Kuijlaars and Vanlessen [23]. They actually established 
the result not just for the usual Jacobi weight, but for a modified Jacobi weight. Adapting 
their notation to ours they show, using the Riemann-Hilbert method, that 

/ Q+l/2 a+l/2\ 

K^^)(x,i/) = B^{x,y) + O I 1 J as n ^ oo, (54) 

where the error term holds uniformly in x and y on bounded subsets of (0, oo). From the de- 
composition PrB^Pr = XR^aYR,a it also follows easily that \Ba{x,y)\ < Cx"~^^^'^y"~^^^'^ uni- 
formly in X and y on bounded subsets of (0, oo). These estimates imply that PrKIt^'^^ Pr — 
PrB^Pr weakly and tTace{PRK^'^^ Pr) — )■ iia.ce{PRBaPR) as n — i- oo. Because for real a 
and /3, the operators K^n'^^ and B^ are positive, using [351 Thm. 2.20], one can conclude 
that PrK^^^Pr ^ PrB uPr in the trace norm. □ 

The question arises whether the restriction to real a, /3 in (ii) is necessary. We conjecture 
that it is not. It seems likely that one can prove estimate fj5^ by repeating the arguments 
of [23]. Moreover, one could perhaps also prove (15^ by using the classical results on the 
asymptotics of the Jacobi polynomials. 
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Assuming the validity of this estimate it follows immediately that the traces converge and 
that PRkt'^^Pn PrB^Pr in the Hilbert-Schmidt norm. Using regularized determinants 
the validity of flS^ would follow for complex a. 

It is probably true that even PrK^i^'^^ Pp> PrB^Pr in the trace norm for complex a. 
Since the operators are not positive definite the argument would be more complicated. This 
statement is dispensable for our purposes, and hence we will not discuss this issue further. 

The restriction to real a, /3 is only temporary. It will be removed at the end of Section [6] 
using an analyticity argument. 

Proposition 5.2 Let R > be fixed, and assume that a, (3 > —1 are real. Then 

j2 ^ det ( /_-|^ (ix 



P(") [R) = exp ( - ^ + aR) ^lim j- (55) 

det ( / , x3^^w{x) dx ) 



n-l 
,k=0 



with 



and 



Pn = ^--r^, Pn = = 1 + 0{n ). 

Wnix) = {1 + fjl-2pnx)''{2 + 2xf, = (2 - 2,t)"(2 + 2x)^. (56) 



Proof. Suppose that 2n > R. From (I5T1) and (1521) it follows that we can decompose 

Kn°''^^\L2[o,R] = AB, where 



A : {x,r,zl eC-^fe L'[o,R], fix) = - 

k=0 



and 



Jo 

Recall that w{x) = (1 — x)"(l + x)^. Then BA is an n x n matrix 

^2 \ 



n-l 



\ n—1 

Pj{y)Pk{y)w{y)dy\ 

'2^ / j,k=0 
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Hence (using the orthonormality of the pkS and the formula det(/ — AB) = det(/ — BA)) 
det{I - ki^-^^)\mo,R] = detl r ^ p,{y)pk{y)w{y)dy 



1 



j,k=0 



This equals 

'n— 1 \ / „i^JL 



Y[(rl] det I j ^ x^+''w{x) dx 



where ak is the coefficient of the leading term of Pk{x). Obviously, 

JJ det / x^+^wix) dx 
From this we can eliminate the product of the cr|'s, and we arrive at 



^ p2 \ n-l 



det I ^ x^~^'^w{x) dx 

det(/ - Kt''^)\mo,Ri = \ ^ ^4^. (57) 

det ( x^^^w{x) dx 



j,k=0 

In the integral appearing in the numerator we make a substitution a; = p„(?/ + 1) — 1 with 
ML 

>1 



Pn = 1 - ^ to obtain 



P^'^' / {y + lny^^w{prXy + l)-l)dy 



-1 



7n = 1 — l/p„. By expanding (?/ + ■jn)-^{y + In)'' using twice the binomial formula and by 
performing row and column operations, the corresponding determinant equals 



det ( /\^+MPn(2/ + 1) - 1)) ^^y) 



The new weight therein evaluates to 

(2 - p„ - PnyTp^iy + ly = ((1 + P?J/2 - PnyTiy + 

where (1 + p^) / (2p„) = (2 — p„) / p„ and p^ G (0, 1). Thus the determinant in the numerator 
of ([ST]) equals 



Pn Pn^'^^Vn"" ^Ct / X^+'^((l + p^)/2 - p„x)"(x + if dx 



n-l 



1 / j,k=0 
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This is almost the same expression as in the numerator of (155|) except for a factors of "2" 
in the weight. As the denominator in ( 157|) with the weight w{x) lacks the same factor as 
compared to the weights w{x), the corresponding quotient is the same. Now it remains to 
remark that the limit of Pn""^^^ l^n"""" as n — )■ oo gives the constant exp(— i?^/4 + aR). 
This completes the proof. □ 

We could, of course, compute the determinant appearing in the denominator of fl55l) 
explicitly. It is, up to a factor 2^°+^)'^, just the product of the cr^^'s appearing in the proof, 
which are known quantities for the Jacobi polynomials. We could see that the product is 
nonzero, and hence the determinant in the denominator of (l55l) is nonzero, too. However, 
this computation will not be of help for us since we need this expression in order to perform 
some kind of cancellation with the determinant in the numerator. 

Theorem 5.3 Let R > be fixed, and assume that —l<a,P<l are real. Then 

R^ 



P^'')(R) = exp ( - — + aR] lim \ 

^ 4 



where 



V'n(t) = ( ) M-l/2,lWwi/2-/3,-l(^), 



(58) 



1- Pnt \ 

ij(t) = U^l/2-a,l(t)ui/2-p-l{t), teT, 

and fin G [0, 1) is a sequence such that fin = ^ — R/n + 0{l/n'^) as n oo. The expression 
under the limit in 15^) is well defined for each n. 



Proof. Using the notation (14311 for the Hankel determinant and the weights (1561) it is easy 
to see that 

det ^ J^^ x^^^Wn{x) dx ^ 



n-l 

j,k=o _ deiHn[wn] 



det ( f^, x^~^''w(x) dx] "^^^ Hn[w] 

V^^ J i.k=o 



j,k=0 

Notice that a common factor due to the particular scaling of the moments ( l44l) cancels 
out. Now we only need to apply Theorem 14.21 twice in order to express the two Hankel 
determinants as determinants of the other type. 

We also remark that the operators I + H{tpn) and I + H{iIj) are invertible due to Theorem 

12.11 Moreover, the determinant det {^n{I + H{ip))~^Pn^ is nonzero for all n. This can be 

seen by relating it to the determinant in the denominator of (1551) . which is nonzero as has 
been pointed out above. □ 



32 



6 Some asymptotic analysis 

The goal now is to identify the hmit of 

det (^Pn{I + H{^n))-'Pn) 

det (p„(/ + if(^))-iP„) 

as — 7- oo. Although the functions ipn and ip depend on /3, in view of (|58ll the limit is 
independent. Hence it is sufficient to do the analysis for one particular value of /3. It turns 
out that the choice /3 = —a makes things sufficiently simple. It is also possible to do the 
analysis for arbitrary /3, but the resulting formula is the same. 

In case (3 = —a, the functions ipn and ijj defined in Theorem 15.31 become 

i^id^ ( ^e^ ( iB^ r e^("+^/2K if o < < tt 



—a 



i^nit) = [ ) U^l/2,l(t)ui/2+a-l(t). (60) 



and 

7/,„ (f^ = ( 

l-flnt-\ 

For the purpose of the following lemma, recall the definition of the operators and 
given in fl39|) and fHOj) . Clearly, is invariant under G^, i.e., G^ip = ijj- In order to rewrite 
ipn notice that 

^ " ^ " ' -t)- f-^^y = • (G-Vi)(t). 



Here the principle values of the power functions are considered. From this we conclude 
ipn = ■ {G~^Ua,i)- Now we define 

^ := G^,>n = ■ = M-1/2,1 " Ul/2+a-l- (61) 

Remark that the sequence /i„ used in these above formulas satisfies 

R 

Hn = l- — + 0{n''^), n oo, (62) 
where i? > is fixed. Finally, let us define the function 

/iR(t)=exp(i?^), teJ, 
and the operator = HiJiR)'^ acting on iJ^(T). 
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Lemma 6.1 Let R > be fixed, \Rea\ < 1, and let ipyipn, and ip he defined by /f5P|). / fgOj) . 
(EIP, and [6^) . Then the following holds. 

(a) The operators I + H{ip) and I + H{'ij)) are invertible on H'^(T), and the sequence 
{I + H{i)n)}nez+ stable. 

(b) We have, as n ^ oo, 

R,,^ + i/(^))-ip„)i?;„ + R,ARl ^ n^(/ + HW-')Un + (/ - Un) (63) 
strongly and 

R^^p^ ((/ + H{^^))-' - (/ + Hi^p))-') Pr^R;^ ((/ + Hm^^ - (/ + 

(64) 

in trace norm. 

Proof, (a): The invertibility of I + H{ip) and I + H{i^) follows from Theorem 12.11 Moreover, 
the operator 

/ + Hiijn) 

is invertible for each n > 1 and the inverses are uniformly bounded. Indeed, this can be seen 
by making a unitary transform, 

R,^Hi^P^)Rl=HiG,„^r.)=Hi^), 

using formula fill I) . For the inverses we obtain the formula 

(/ + if(^„))-^ = r;ji + Hm-'R,^, 

from which the uniform boundedness of the inverses follows. 

(b): Let us start with some preliminary considerations. Observe that P„ = H{t'^Y and 
that 

^i^,n:=G,„(r)=f-^y, 

which implies, by (|4T1) . that 

Rf,^PnR*^„ = H{hR.n) ■ 

The functions Hr^u are uniformly bounded in the L°°-norm, and using the asymptotics (1^^ 
it is easily seen that hji n in measure as n — )■ oo. From Lemma [3.31 we can conclude 

that 
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strongly on H^{T). Moreover, because the functions h^^n and all of its derivatives converge 
locally uniformly on T \ {—1} to the function and its corresponding derivatives, we can 
conclude that 

HihR^J) ^ Hihnf) 

in the trace norm whenever / is a sufficiently smooth function on T which vanishes identically 
in a neighborhood of t = — 1. 

Furthermore, from the definitions it follows that 

Combining all this we conclude that the first expression to analyze equals 

H{hR^n?{I + H{^))-'H{hR^^f + 1- H{hR^^)\ 

and that this expression converges strongly to 

Hih^fil + Hm-'H{hnf + / - H{hnf. 

This is equal to the right hand side. By definition HihR)"^ = Ur. 
The second expression to analyze equals 

H{hn,n? ((/ + Hiij))-' - (/ + Hiij))-') H{hn,nf. 

Now choose two smooth, even functions /i, /_i such that /i + /_i = 1 and such that /i 
vanishes on a neighborhood of 1, while /_i vanishes on a neighborhood of —1. Then the 
above expression equals the sum 

H{hn,n?nf_^) ((J + Hiij))-' - (J + Hiij))-') H{hn,n? 

+ H{hR^^fT{h) ((/ + Hm-^ - (/ + H{hR,,)\ 
In view of the first term, we write (see 

H{hn,n)T{f-i) = H{hR,J^,)-T{hn,n)H{f^,). 
This is trace class and converges in the trace norm to 

H{hR)T{f_,) = Hihnf-i)-Tihn)Hif_,). 
As to the second term, consider 
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Hence T{fi)H{ip) equals H{%lj)T{fi) plus a trace class operator. From this we can conclude 
that 

T(/i)(/ + H{iP))-^ = (/ + H{^))-^T{fi) + trace class. 

Next observe that fii^ip — "ip) is continuous and has one-sided derivatives at t = —1, while 
it is smooth elsewhere. Hence H{{'il) — ip)fi) is a trace class operator, which implies that 
T{fi)H{tp) equals T{fi)H{ip) plus a trace class operator. Using the above it follows that 
H{ip)T{fi) equals T{fi)H{ip) modulo trace class. This implies that 

(/ + H{ij))-^T{fi) = T(/i)(/ + H{ij)y' + trace class. 

Combining with the above we see that 

T{f,)(^{i+Hm-'-{i+Hm-') 

is trace class, and from this the desired conclusion follows easily. □ 

The operator Ur is a projection, i.e., H^ = H^^. Indeed, we have hu G H°°(T) and 
JiR = h^' E H°-{T). Hence by (EHl) we have H{hn)Tihn) = HQirKr) - T{hR)H{hR) = 0, 
whence by (^^, 

Hihnf = H{hR){I - T{hR)T{hR)) = H{hR). 

This shows that H{hR)^ is a projection. Let us denote the image of Hr by "imH/j", which 
is a closed subspace of if^(T). 



Theorem 6.2 Let R> be fixed, \Rea\ < 1, and let ip,'ipn, and ip he defined by (5U) . (6(a) . 
(61\) . and (EE)- Then the operator 



URii + H{tp)r'nR 

y / imrij: 

is invertible on the space imH/j, and 

det (p„(/ + iJ(^„))-ip„ 

lim y 

det (Pn{I + Hm-^P^ 

= dei\(uR{I + H{i,))-^TlX\uR{I + H{i,))-^Tlj^ 1. (65) 
In particular, the expression on the right hand side is well-defined. 

Proof. The expression for which we want to determine the limit is equal to the determinant 
of 
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By Lemma [6.1( a). the inverses of / + H{iIj) and / + H{il)n) exist for each n. Moreover, by 
Corollary 13.121 the sequence {Pni.1 + H Pn} n&+ is stable. In fact, we can even say 
that the inverses of these finite sections exist for each n as noted in Theorem 15. 3[ Hence the 
above expression makes sense for all n. 
We rewrite the above expression as 

/ + (P„(/ + H{ij))-'Pn + Qn) ' (Pn ((/ + i/(^n))"' " (/ + Pn) 

and multiply with the unitaries i?*^ and , which does not change the value of the deter- 
minant, in order to obtain 

By Lemma 16.1( b) the expression on the right hand side tends to 

((J + Hii,))-' - (/ + Hii;))-') Un 
in the trace norm. Since the sequence Pn(/ + H{tp))^^Pn is stable, so is the sequence 

p;„p„(j + i/(^))-ip„p^„ + p;„g„p^„. 

This sequence converges strongly to 

UR{I + H{ij)r'Un + {I-Iln). 

This last operator is also invertible. This follows either from Lemma [3nT ii). or from Corollary 
13.61 Combining all this, we conclude that the limit under consideration exists and equals 
the determinant of 

/ + (n^,(/ + H{^p))-'UR + (/ - n^))"' (n^ ((/ + H{ij))-' - (/ + if(^))-i) n^). 

This operator equals 

(n^(/ + H{ij)r'Un + (/ - n^)) (n^(/ + H{ij))-'Un + (/ - n^)) , 

the determinant of which is the same as in (jHSD- n 

We remark that the determinant on the right hand side of fl65|) cannot be written as 
the quotient of two determinants of the corresponding operators because (as one can show) 
neither of the operators 

(iIr{I + H{^)y'Un) and (n^(/ + H{ij))-'UR^ 
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is of the form identity plus trace class. The reason is that the generating functions of the 
Hankel operators have a jump discontinuity at t = — 1. 

Let us now state the main result of this and the previous section. It concludes Step 1 
discussed in the introduction. 



Corollary 6.3 Let R> and \Rea\ < 1. Then 



+ aR I det 



-1 



Pjiil + H^Wy'Pn) iPRil + H^m-'PR) , (66) 



P(")(i?) = exp 
where 

1p{x) = U-l/2-a,o{^)Ul/2+a,oo{x), = U-l/2,o{x)Ui/2+a,oo{x) . 

Proof. By Corollary 12. 2[ the operators / + H]^{iIj) and / + H]^{iIj) are invertible on L^(]R+). 
Moreover, the operator Pii{I + H]s^{iIj))~^ Pf> is invertible on L^[0,i?] by Corollary 13.61 In the 
previous theorem we have shown that 



A 



-1 



imnfl. 



(with i/) and as defined by ( 159|1 and ( l6Ti) ) makes sense and is of the form identity plus trace 
class. We apply the transform S to A and obtain that 



SAS-' = (Pr{I + H^{4j))-'Pn] (Pr{I + H^Wr^PR 



which thus is also of the form identity plus trace class. Hence the determinant in m&i is well 
defined and equals the right hand side in fl65|) . Notice that here we have used 

SH{a)S'' = H^{a), SUrS'^ = Pr, 

see f l3T]) and f p2|) . where the last identity follows from the fact that Pr = H^{e^^^)'^, and 
SH{hR)S-' = ifM(e-«). 

Combining this with the previous theorem and Theorem 15. 3[ it follows that the identity 
OnH) holds for all real a with \a\ < 1. 

Now, by Theorem 15. II the quantity P^"\R) depends analytically on a for |ReQ;| < 1. The 
same is true for the right hand side of (166|) since the generating functions ip and ip depend 
analytically on a. Hence the identity (!66|) holds not just for real a, but for complex a. This 
completes the proof. □ 
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7 Localization 



Let B stand for the Banach algebra of all function a G L°°(T) for which both H{a) and H{a) 
are trace class (see [lOl Sec. 10.2-3]). The norm in B is defined as 

Wo-Wb = kol + \\H{a)\\ci{H2{T)) + ||-f/'(a)||ci{H2(T)). 

The class B can be identified with the so-called Besov class Bl, however, we will not make 
use of this fact. What is important for us is that B contains all smooth functions as a dense 
subset and that the Riesz projection is bounded on B. Using this and Gelfand theory, it 
follows that the maximal ideal space of B can be identified with T. In other word, a & B is 
invertible in B if and only if a{t) ^ for all t G T. Moreover, a G i3 possesses a logarithm 
in B if and only if a possesses a continuous logarithm. Now we define the unital Banach 
algebras 

B^ = BV\H'^{J), B^ = Br]H°°{T), 

and we can introduce the notion of Wiener-Hopf factorization in B similar as for W (see 
Sec. S]). We also recall that GB stands for the group of all invertible elements in a unital 
Banach algebra B. 

What is also important to us is that B contains all continuous functions on T which are 
smooth except at finitely many points at which the one-sided derivatives exist. This is easy 
to prove using the fact that the Fourier coefficients of such a function decay as 0(n~^). 

We start with the following result, which has essentially been proven already in [2]. We 
will sketch the main idea of the proof in order to indicate how the constant arises. For more 
details we refer to [2]. 

Proposition 7.1 Let 6+ G GB+. Then 

det(/ + i/(6+6;^))= (^i^y^^xp(-igMlogM^)- (67) 

Proof. Let b = b+b+. Then T{b) = T{b+)T{b+), whence T-\b) = T{b-^)T{b^^), and 

T-\b)H{b) = T{b-')T{b-')H{bM = T{b-')H{b^). 

Using H{b+)T{b^^) = H{b+b^^), it follows that 

det(/ + T-\b)H{b)) = det(J + H{bJ)-^)). 

Here we have used the formulas (p71) . In [2, Thm. 2.5] it was shown that det(/ + T^^(6)/f(6)) 
equals the right hand side in (167|) . The crucial point is to introduce an operator M(a) = 
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T(a) + H{a) and to observe that M{aia2) = M{ai)M(a2) whenever 02 = 02. Indeed, this 
last identity follows from and fl25|) . 

Now consider the function b\ = exp(Alog6), which depends analytically on A, and define 
the analytic function /(A) = det T~^{bx)M{b\). Take the logarithmic derivative of / (by 
employing the formula (logdetF)' = trace(F'F~^)) and differentiate once more. A simple 
computation yields 

= trace (M(6',)M-1(6a) - T-\bx)Tib'^)) = trace (M(log6) - T'\bx)Tib'x)) 

and 

' ' trace {T'\b,)Tib\)T-\b,)Tib',) - T-\b,)Tib';^)) 



Now we use the facts that b'^ = bx log b and 6" = bx log^ b and that 6a has a canonical Wiener- 
Hopf factorization in B, say 6a = bx,+bx- with 6a, ± G GB±. Using T~^(6a) = T(6^^)T(6^^) 
and (1271) we obtain 



V/(A) 



trace (T(log6)T(log6) - T(log2 6)) = -trace {H{\ogb)H{\ogb)) 



which does not depend on A. Integrating and fixing the constants with the values of / and 
/' at A = yields 

/ A^ 
/(A) = exp ( A trace (log 6) — ^trace (if(log6)if(log6)) 

From this the assertion follows by putting A = 1 and by evaluating the traces. □ 
We proceed now with two rather technical lemmas. Let us recall the notation Gr defined 

in am. 



Lemma 7.2 Let a = Ua,i, c = u-y^i — 1, and iprit) = (1 — t)a(t){Grc){t) for t G T, r G [0, 1). 
Then H{tpr) is trace class and tends to zero in the trace norm as r — )■ 1. 

Proof. We first remark that it is rather easy to see that ipr converges uniformly to zero on 
T. In fact, the function GrC converges to zero locally uniformly on T\ {1} and it is uniformly 
bounded on T. Hence the multiplication with (1 —t) implies the uniform convergence on all 
of T. As a consequence the Hankel operator converges in the operator norm to zero. The 
technical difficulty is the convergence in the trace norm. 

For fixed r, the functions ipr are smooth on T \ {!}, continuous on T, and have one- 
sided higher order derivatives at t = L This implies that all the Hankel operators are trace 
class. We will soon use the fact that the functions (1 — t)a{t) and (1 — t)c{t) have the same 
properties. Hence the Hankel operators with these symbols are trace class, too. 
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In order to show the convergence of H{il)r) in the trace norm use (125|) and write 
H{il)r) = T((l - t)a)H{Grc) + H{{1 - t)a)T{G^c). 

The sequence GrC converges in measure to zero and is uniformly bounded. Hence the adjoint 
of T{Grc) converges strongly to zero. On the other hand, the Hankel operator H{{1 — t)a) 
is trace class. It follows that the second term in the above expression converges to zero in 
the trace norm. Hence our concern from now on is the first term. 

Applying the unitary operator Rr defined in (HOi) . the trace norm of this operator is equal 
to the trace norm of 

i?;T((l - t)a)H{Grc)Rr = R;T{{1 - t)a)RrR;H{Grc)Rr 

= T{G;\{l-t)a))H{c). 

One computes easily that 

. (1 - - 

1 + rt 

Because of (1271) we conclude that 



T{G;\{1 - t)a))H{c) = T — - G;'a] T(l - t)H( 

\l + rt J 



c). (6^ 



Observe that | (1 — r)/(l + rt) | < 1 and that (1 — r)/(l + rt) converges locally uniformly on 
T \ {—1} to zero. Hence this term converges to zero in measure. The sequence G~^a is also 
uniformly bounded. We conclude that the Toeplitz operator on the left of the right hand 
side of the last equation tends strongly to zero. Using (!25|) now write 

T(l - t)H{c) = H{{1 - t)c) - H{1 - t)T{c) 

to see that this operator is trace class. We conclude that the expression f lBSl) converges to 
zero in the trace norm. This finishes the proof. □ 

Let us define the expression 

K{a, b) = {I + H{ab)) - (J + H{a)){I + H{b)). (69) 

for a,6 G L°°(T). 

Lemma 7.3 Let a = Ua.i, b = and 

Then K{a, b) is trace class, and K{ar, K) — )■ K{a^ b) in the trace norm as r — )■ 1. 
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Proof. We first observe that we can write K{a, h) as 

K{a, b) = H{{a - l){b - 1)) - H{a)H{b). 

The function (a — — 1) is continuous and sufficiently smooth on T \ {±1} and has 
one-sided derivatives of arbitrary high order at t = ±1. Hence the Hankel operator H{{a — 
l){b — 1)) is trace class. To see that H{a)H{b) is trace class, decompose 1 = /i + /_! into 
smooth and even functions such that fi is identically zero in a neighborhood of 1, while /_i 
is identically zero in a neighborhood of —1. Then write, using fl25|) . 

H{a)H{b) = H{a)T{h)H{b) + H{a)T{f_{)H{b) 

= {H{ah) - T{a)HU\)) H{b) + H{a) {HU-ib) - H{f^,)T(b)) , (70) 

which is trace class because so are the Hankel operators with the symbols a/i, f-ib, fi, 

The functions and br are smooth. Hence the Hankel operators H{{ar — l)(&r — 1)) and 
H{ar)H{br) are trace class for each r. Let us first show that 

H{ar)H{br) ^ H{ua,i)H{u^,^,). 

We use the identity (1701) with a,, and br instead of a and b, and hence we have 

H{ar)H{b,) = {H{arh) - T{ar)H{h))H{b,) + if (a,) {H{f^A) - H{f_^)T(br)) . 

Now remark that a^. and br are uniformly bounded and converge in measure to a and b. 
Hence we have following strong convergences, 

T{ar) T{a), Hibr)* ^ H{b)\ H{ar) H{a), T(6,)* ^ T{by. 

On the other hand, a^/i — )■ a/i and 6r/-i ~^ bf-i, e.g., in the norm of C2(T), whence it 
follows that H{arfi) — )■ H{afi) and H{f^ibr) — ?■ H{f-ib) in the trace norm. Combining all 
this we can conclude that H{ar)H{br) H[a)H{b) in the trace norm. 

In order to treat convergence of the Hankel operator H{{ar — l)(6r — 1)) we use the above 
functions fi and and decompose 

H{{ar - l){br - 1)) = H{{ar - l)h{br - 1)) + H{{ar - - 1)). 

Without loss of generality it suffices to consider the last term. In fact, the first term on 
the right can be transformed into the same kind of expression by the unitary operator 
F_i : fit) ^ f{-t), t e T. Now write, using ([2SD, 

H{{ar - - 1)) = H{{ar - 1))t(/_i^^^) +T((a. - l){t-l))H [fj-^) 



42 



Let us first focus on the terms containing br. Let K be any compact subset of T \ {— 1}. 
We may think of the functions br{t) and b{t) as being defined and analytic on a suitable open 
neighborhood of C C. On this neighborhood of K, we have uniform convergence br — ?• b. 
Because 6^(1) = ^(1) = 1, we have also uniform convergence 

br{t) - 1 b{t) - 1 
t-1 ^ t-1 

along with all derivatives. It follows that 



t-lJ V t-1 

where the convergence of the Toeplitz operators is in the operator norm and the convergence 
of the Hankel operators is in the trace norm. 

Next observe that a,. — )■ a locally uniformly on T\{1}, whence it follows that {t—l)ar{t) — t- 
{t — l)a{t) uniformly on T. Hence 

T((a,-l)(l-t))^T((a-l)(l-t)) 

in the operator norm. In order to treat the Hankel operators with the same symbols write 

Now Lemma [7.21 implies that H{{t — l)ar) — )■ H{{t — l)a) in the trace norm. Combining all 
the previous considerations, it follows that 

ff((a, - - 1)) ^ Hiia - l)f_,{b - 1)) 

in the trace norm, which completes the proof. □ 

Proposition 7.4 Let a = 'U-i/2-a.i; b = Mi/2+/3,~1; O'lT'd assume |Rea| < 1 and |Re/3| < 1. 
Then the operator determinant 

det ((/ + H{a))-\I + H{ab)){I + H{b))-^^ (71) 

is well defined and equals 2~(V2+a)(i/2+/3) ^ 

Proof. Theorem 12.11 implies that the inverses of / + H{a), I + H{b), and / + H{ab) exist. 
Defining K{a, b) as in (!69|) we can write 

(/ + H{a))-\I + H{ab)){I + H{b))-^ =/ + (/ + H{a))-^K{a, b){I + H{b))-\ 
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and hence this operator is of the form identity plus trace class. Its determinant is well 
defined. 

In order to compute the value of this operator determinant, we approximate the functions 
a and h by smooth functions 

1 - rt \ , , , / 1 + rt ^ 



and then let r — )■ 1. 

The sequences and hr are bounded in the L°°-norm and converge in measure to the 
functions a and 6, respectively. This implies that H{ar) — ?■ H{a) and H{br) — )■ H{b) strongly 
on if^(T) as r — )■ 1, and the same holds for the adjoints. Theorem 13.131 implies that the 
sequences {I + H{ar)}re[o,i) and {I + H{br)}relo,i) a-^e stable. Hence using Lemma EHJ^iii) it 
follows that 

{I + H{ar))-^ ^iI + H{a))-^ and ((J + ^ ((/ + ^(&))"')* 

strongly on if^(T). This together with Lemma [7.31 implies that 

(/ + H{ar))-'K{ar, hr){I + H{h,))-^ ^ (/ + H{a))-'K{a, h){I + H{h))-^ 
in the trace norm as r — t- 1. Hence the determinant 

det ((/ + H{ar))-\I + H{arhr)){I + i^(&r))"') (72) 

converges to (17T1) as r — )■ 1. In (1721) each of the Hankel operators is trace class, and hence we 
can split it into the pro duct /quotient of three determinants, each of which we can evaluate 
by Proposition 17.11 We obtain 

(7 \ \ 1/2 ^ oo 



(CL (l)b (1) \ 1 
Q^"Y-1)C(-1) / ^ ~ 2 ^ ^^'^ ^ 



2 

rjfc 



Here a+,r(t) = (1 - rt)-^/^-^^ = (1 + rt)i/2+'^. Hence the determinant (1721) equals 

exp ^-^fc[loga+,,.]fc[log6+,^]fej = exp ^7 ^ /c ^-^^ 



fc=i 



2^-7 
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with 7 = (1/2 + a)(l/2 + f3). Now take r 1. □ 

Proposition 7.5 Le^ a = U-i/2-a,o, b = ui/2+13,00, and assume that \Rea\ < 1, |Re/3| < 1. 
Then we can write 

Pr{I + H^{ab))-'PR = Pr{I + H^{h))-^Pn{I + H^{a))-^PR + PrKPr + Cr, 

where K is a trace class operators on L^(]R+) and Cr are trace class operators on L^[0,-R] 
tending to zero in the trace norm as i? 00. 

Proof. Let us first remark that the operators / + H]^{a), I + H^^b), and / + H^(ab) are 
invertible because of Corollary 12.21 From the part of Lemma (7.31 which states that K{a,b) 
is trace class (see also (!69|) ) it follows that 

(J + Huiab)y' = (/ + HRib))-\l + H^ia))-^ + K, (73) 

where K is a trace class operator. To see this we have to apply the transformation S (see 
dSH) and ([32D). 

Let V±R = W{e'^'^^^). These operators are the forward and backward shifts, 

= { ^^"^q" i[ Q ^ f < {V-Rf){x) = fix + R), x> 0. 

Clearly, Qr = VrV-r, and the formula V-rH^{c) = H^{e~^^^c) = Hr{c)Vr holds, which 
follows from the continuous analogue of (1271) . Using the identities 

(/ + B)-^ = /-(/ + B)-'B, (/ + A)-' = I-A{I + A)~' 

with B = H^{b) and A = H^{a), it follows that 

Pr{I + H^{b))-'QR{I + HMr'Pn = Pr{I + H^{b))-^H^{b)QRH^{a){I + H^{a))-^PR 

= Pr{I + H^{b))-'V.RH^{b)H^{a)VR{I + HMY'Pr- 

This term converges to zero because H^{a)H^{b) is trace class and — )■ strongly as 
i? — 7- 00. Combining the previous formulas, using I = Pr + Qr, the desired formula follows 
easily. □ 

Theorem 7.6 Let a = U-i/2-a,o, b = ui/2+13,00, and assume that \Rea\ < 1, |Re/3| < 1. 
Then 



lim 



det 


(^PR{I + H^{b)y'PR^ 


'(^PRiI + H^{ab)r'PR^ 




det (PR{I + H^ia))-^PR) 





2(l/2+a)(l/2+/3) 



All the expressions on the left hand side are well defined for sufficiently large R. 
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Proof. Because of Corollary 13.91 the sequences 

{PR{I + H^{a))-'PR}R^o and {Pr{I + H^{b))-'PR}Ryo 

are stable. Moreover, taking the strong limit on L^(R+) as i? — )■ cxd we get (/ + H^^a))"^ 
and (/ + H^{b))~^. Similarly, we can take the strong limit of the adjoints. 
For sufficiently large R it is thus possible to consider 

(^Pr{I + H^{b))-'PR^ (Pk(/ + Hu{ab))-'PR^ (^Pr{I + H^{a))-'PR^ ~\ 
which, by Proposition 17. 5^ equals 

Pr + [Pr{I + H{b))-'PR^ '^k{Pr{I + H{a))-^Pj^ ' + Cr 



with {CR}Ryo being a sequence of trace class operators on L'^[0,R] tending to zero in the 
trace norm, and with K being trace class operator on L^(R_|_). Complementing with the 
projection Qr, which does not change the value of the determinant, we can rewrite this 
identity as 



Qr + [PRil + Huib))-'PR^ ' (Pk(/ + Hu{ab))-'PR^ (Pi^(/ + H^ia))-'PR^ 

= /+(/ + H^{b))K{I + H^ia)) + Dr. 

with a sequence {-D/j}h>o of trace class operators on L^(R+) tending to zero in the trace 
norm. Using the expression of K given in (175]) . which can also be obtained by passing to the 
limit i? — )■ oo in the previous equation, it follows that the above equals 

(/ + H^{b)){I + H^{ab))-\I + HM) + Dr. 

It follows that the following operator determinant is well defined 

-1 / \ / \ -1 



det 



Pr{I + H^{b))-^PR^ (Pr{I + H^{ab))-^PR^ {Pr{I + H^{a))-^PR 



and that its limit R ^ oo equals 

det ({I + H^{b)){I + H^{ab))-\I + HM) 



Applying the transform S (see fl?I]) and fl5^ ) and noting that the reciprocal of resulting 
determinant has been computed in Proposition 17. 4[ this completes the proof. □ 



46 



8 The final result 

Let us now put all the pieces together and derive the final result. In Corollary 16.31 we have 
shown that for |ReQ;| < 1 and i? > we have the identity 

i?2 



P(")(i?) =exp - — + X 

^— 1/2— a,0^1/2+a,oo 

Therein the operator determinant is well defined (see also Corollary 13. 6p . For sufficiently 
large R, this determinant can be written as a product of the following two determinants, 

1/2— Q,0^1/2+Q,oo 
^l/2+Q,oo ))-'Pr) (^Pr{I + H^{ l/2,0^1/2+Q,oo 



In this connection observe that Corollary 13.91 guarantees that the inverses of the various 
operators PR(/+if]R(*))~^PR|2,2[o,H] 6xist. In particular, dk{R) 7^0. Notice also that Theorem 
17.61 makes sure that both operator determinants are well defined. Moreover, Theorem 17.61 
implies that 

, , det(Pn{I + H^{u^^/2,o))-'PR 



det (^Pr{I + HR{u^i/2-a,o))-'PR 

as i? — 7- oo. To complete the argument we need a (non-trivial) result established by E.L 
Basor and the author in [5l Sec. 3.6]. Therein G{z) stands for the Barnes G-function ([6] 
(see also P). 

Theorem 8.1 Let -3/2 < Re7 < 1/2. Then, as R-^ oo, 

G{l/2) 
G(l/2-7) 



det [Pnil + H^iu,,o))-'PR) ~ p7V2+7/2(2^)-7/22-7--7/2 ^^^lV^J^^ _ (74) 
Applying this theorem we obtain 



/^(a2-l/4)/2(^27r)"/2+l/42-«("+l/2) G'(l) ' 

Thus, after simplifying, we get our final result, which confirms the conjecture of Tracy and 
Widom [39]. 

Theorem 8.2 Let |Rea| < 1. Then, as R ^ oo, 

p(")(fl).exp(-^ + „fl-^logfl)^^. (75) 
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